Light is often used in order for an individual to see a surrounding environment. While sunlight provides an illuminated environment, several situations exist where sunlight is not available or may not be desirable. In such situations, some form of illumination or vision enhancing device is necessary. For example, candles, light bulbs, and flashlights have been historically used as illumination devices. Conversely, night vision goggles and other infrared devices have typically fulfilled the need for a vision enhancement/imaging device. As technology has advanced, an increasing number of methods and devices for viewing our environment have been developed.
(1) Heads Up Display (HUD)/Head Mounted Display (HMD) Approaches
In order for a user to view a surrounding environment, several HUD and HMD tools have been devised. For Example, the University of Pittsburgh has devised a sonic flashlight for blending ultrasound images with physical objects such as a human hand. An ultrasound transducer illuminates the object. A display is used on the opposite side of a half silvered (translucent) mirror from the transducer and reflects an optical image onto the object. This relies on precise geometric relationships amongst the ultrasound slice being scanned, the monitor displaying the slice, and the mirror. However, the University of Pittsburgh device does not contemplate using the world as a display surface.
Traditional HUD-based display of night vision imagery performs no processing on the image to enhance perception and does not project the image onto the environment of the user, but rather onto the optical elements that make up the HUD. Traditional “video see-thru” Augmented Reality systems are similar in that they use a sensor, typically a charge coupled device (CCD) camera, to capture images of the local environment and then display those images on a head worn display. This requires sophisticated tracking systems to be worn by the user and does not anticipate using the world around the user to project information.
Furthermore, there are two projective HMD systems that have been developed in research labs. One is from Susumu Tachi's group, and the other from Jannick Rolland and Hong Hua at the University of Central Florida. Both systems have the user wear a projector on his head. The projector illuminates parts of the environment covered in retro-reflective material, so the projection bounces directly back to the user's eyes. However, the projector is not bright enough to work without the retro-reflective surface and the ambient brightness must be low. In one demonstration, the Central Florida system has a separate optical tracker measuring the locations of “Go” pieces on a “Go” board. The system can then augment the real board with virtual “Go” pieces. However, these approaches require external head trackers and do not capture a video image of the real environment through a head-worn video camera nor do they process and project such an image.
(2) Projection Systems/Lighting
Projection and lighting systems have been employed to illuminate an environment. For example, handheld devices with portable projectors are made by several companies. The portable projectors are not integrated with any sensing or processing means and do not supply sufficiently focused light output to be used in environmental projection.
As another example, Oregon Scientific, located at 19861 SW 95th Place, Tualatin, Oreg. 97062, sells a projection alarm clock (Model Number: RM-313) that automatically synchronizes to the United States (US) Atomic Clock and then projects the time in bright red light on a ceiling surface. Although the system possesses the means to sense the US Atomic Clock signal, process it, and display it onto the environment, it does not offer any capability to generate a display of perception enhancing information.
Additionally, a company named “Image Projection”, located at 1109 Ponce de Leon blvd, Coral Gables, Fla. 33134, uses powerful light projection systems to project light images onto projection mediums, ranging from specially crafted projection surfaces to the retail products themselves for advertising. They sell projection systems that use static images (called “gobos”) with optics and bright lights, including a small low voltage projector (3 in×2 in). Although, the product uses a projector to project an image, the product does not include any means to sense and process a local physical environment.
Projection technology has been employed in the construction industry in equipment containing rudimentary capabilities for sensing and displaying limited information. For example, a self-leveling plumb line (the Laser Vision Plumb Pyramid) made by Zircon, located at 1580 Dell Ave., Campbell, Calif. 95008, projects a vertical line onto a wall and will compensate for up to five (5) degrees out of level. Similar tools for finding wall studs and displaying the location on light emitting diodes (LEDs) on the tool also exist. These systems do not contemplate the registration of information with environmental features and do not offer the means to generalize their capability.
Furthermore, the Massachusetts Institute of Technology (MIT) Media lab has done ongoing work in the area of “personal projectors” which focuses on the development of small, low power, hand-held devices that can project information onto surfaces. These personal projectors do not contemplate the use of sensor input or registered projection to enhance perception in their concepts. MIT also describes work in the development of “ambient displays or fixtures” that use the physical environment for display of information. However, their approach does not contemplate enhanced perception and only provides architectural and aesthetic display of information (e.g. a “water lamp” that projects raindrops on the surface of still water). Other Media Lab projects have used projectors as part of interactive games, but without attempting to enhance perception with registered projection of sensed imagery or data.
(3) Laser Lighting/Vector Display
Laser lighting displays used for events such as rock concerts offer the means to project predetermined patterns from a processor into the environment of the audience (typically on smoke in the air). However, the laser lighting displays do not project sensor data onto the environment using laser systems as vector displays. Also, rock concert laser light shows are typically not eye safe (must be no more than 1 Milliwatt) in the manner needed for the present invention.
(4) Structured Light
Structured light sensing uses a light projection system to project a known pattern onto an object. The pattern distorts depending on the shape and depth of the object and using a normal charge-coupled device (CCD) camera, algorithms can capture the three-dimensional (3D) geometry of the object.
Ferreira, Lobo, and Dias describe a handheld scanner using structured light projection called the tele-3D, which aims to produce 3D surfaces for small objects. Rocchini, Cignoni, Montani, Pingi, and Scopigno, (“A low cost 3D scanner based on structured light,” Eurographics 2001, Vol. 20, Number 3) describe a low cost scanner using color-striped patterns to acquire a 3D model of a small artifact. These systems and many more like them are only used in the capacity to determine 3D depth from a scene.
Researchers at the University of North Carolina (UNC) (Raskar, Fuchs, Welch, Lake, and Cutts: 3D Talking Heads: Image Based Modeling at Interactive Rates Using Structured Light Projection) have developed systems that project imperceptible structured light onto surfaces to determine their 3D shape and then project an image onto the surface. More recently, UNC researchers (Raskar, Welch, Cutts, Lake, Stesin, Fuchs, “The Office of the Future: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays,” SIGGRAPH 98) describe a system that uses structured light (potentially imperceptible) to capture depth per pixel, then project images onto the 3D surfaces with correct geometry. The system is auto-calibrating in that it does not require the user to describe the geometry of the display areas prior to projecting imagery. The projector is used for both scanning the surfaces of the environment (using structured light) and for display. However, this system does not anticipate use of the environment for arbitrary display of information from sensors for the purpose of enhanced perception.
(5) Intelligent Headlights/Flashlights/Environmental Lights
Intelligent headlight can be used to adjust the direction of the beam (e.g. point around a corner), or at low speed may be used to make the beam short and wide, or alternatively, at high speed to make the beam narrow and longer. Additionally, some LED headlamps use more colors than conventional high intensity discharge (HID) (xenon) or halogen lights. However, these systems have minimal sensing (yaw or steering wheel turn) of the environment and project unstructured light without regard for specific features of the environment.
An example of an environmental light is an active road reflector. Active road reflectors use LEDs and are visible over 1000 M. However, they do not possess any sensing or processing means, but merely act as a static reflector.
In an attempt to provide a more adaptive lighting, Bavarian Motor Works (BMW), located at Chestnut Ridge Road, Woodcliff Lake, N.J. 07677, has proposed adaptive lighting that connects the driver with things like accelerator pedal push back, alertness assistance, parking assistance and lighting systems. More relevant is the BMW concept of “Pixel Light”, which postulates the use of digital micro-mirror devices (DMD) projection technology to provide anti-dazzle permanent high beam illumination, bright Illumination of road markings, and navigation information on the road surface. However, this concept does not describe how to process, transform, and project information to enhance perception from sensors onto the roadway. Further, they do not address problems of tracking the car with precision to display information on the road surface in registration with features and do not address issues in projection of imagery with sufficient brightness to be visible under adverse conditions.
The following patents address similar capabilities.
U.S. Pat. No. 6,341,884, issued to Leleve, discloses a vehicle headlight that is capable of emitting different types of beams. This is a two-mode headlight for providing a cornering beam and a fog light beam.
U.S. Pat. No. 5,761,243, issued to Kornos, discloses display optimization for night vision enhancement systems as an apparatus for displaying an image from the night vision sensor in an aspect consistent with the horizontal field of view.
U.S. Pat. No. 6,420,977, issued to Corbitt, discloses video-monitoring safety systems and methods. The patent describes a system to present video images from a dangerous location to a driver upon arriving in proximity to the video camera.
U.S. Pat. No. 6,163,755, issued to Peer et al., discloses an obstacle detection system. The system is for detecting an obstacle from a sensor in a predetermined field of view and producing an avoidance signal. The patent was primarily designed for rail tracks.
U.S. Pat. No. 8,429,429, issued to Fohl et al., discloses a night vision system utilizing a diode laser illumination module and a method related thereto. The patent describes an active infrared illumination and detection system for improving night vision of obstacles in roads. However, there is no structured projection of information for enhanced perception onto the road surface and no processing of sensor information to extract relevant features.
As an example of a projector coupled with a micro-computer, Duracell's Watchdog intelligent flashlight has a built-in microcomputer that continuously monitors its batteries and guarantees that they will never go dead on you by automatically disabling the flashlight as soon as the voltage starts to drop. However, the flashlight has no means to sense the environment and does not project information for enhanced perception of the environment.
As another example of an intelligent flashlight, Sony sells an intelligent flashlight with its video recording equipment. The Sony flashlight is configured to deliver an appropriate amount of light for a given scene to be recorded.
U.S. Pat. No. 5,040,007, issued to Hagiuda, discloses an electronic flashlight apparatus for a camera. The flashlight, or flash, can optically direct light from the source in a different direction.
U.S. Pat. No. 4,131,345, issued to Carollo, discloses a visible light projection device. The device produces an image in a variety of angular orientations and sizes using servo controlled prisms.
(6) See-Through Sensors
Motion and Ranging Sensor (MARS) by Raytheon Company, located at 870 Winter Street, Waltham, Mass. 02451-1449, is a radar based system for locating people through walls. The system detects movement and can display the three-dimensional (3D) location of a person or object in several modes, including a plan view and perspective view. However, the system does not anticipate projection of information onto the environment and does not process the 3D sensor data for this purpose.
Another example of a see-through sensor is an ultra wide-band radar. Such a device is capable of detecting breathing up to twenty (20) feet away (through walls). The device then displays the resulting location of the person on a liquid-crystal display (LCD).
Additionally, Georgia Tech is currently pursuing a radar flashlight. The Georgia Tech Radar Flashlight is a flashlight-sized through-the-wall radar with a range of 40 ft. Once a human is detected through motions as small as heartbeats or respiration, the light flashes on a red LED on the handle to indicate that motion has occurred. However, it currently cannot be used while in motion (due to Doppler shifted signals).
As described above, several devices have been employed in order to allow a user to view a surrounding environment. However, each device has a fallback in that they do not sense, process, and project information to enhance a user's perception of an environment. Thus, a continuing need exists for a system that allows a user to gather information from a physical environment and project that information back onto the physical environment in order to provide the user with an enhanced perception of the environment.